Intermittent sampling of storage access frequency

ABSTRACT

The intermittent sampling of storage access frequency is performed by determining a duration of a collection window and a duration of an observation window within the collection window. A position of the observation window within the collection window is randomly selected, and frequencies of accesses of one or more storage objects during the observation window are observed. When a new access of a given storage object occurs, a delta time for the given storage object is calculated as the time of the observed access minus the timestamp of the most recent observed prior access of the given storage object. Optionally, the delta time of two sequential accesses of a given storage object in two different observation windows may be calculated as if the two different observation windows are immediately adjacent to each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application relates to co-pending U.S. patent applicationSer. No. (SVL920130081US1/IBM062), filed on Sep. 4, 2013.

BACKGROUND

Different types of data storage incur different costs, with fasterstorage costing more per gigabyte than slower storage. One approach tomanaging storage costs is to use hierarchical or “multi-temperaturestorage,” where frequently accessed (aka “hot”) data are stored onfaster but more expensive storage (e.g. solid-state “disks”), and lessfrequently accessed data are moved to progressively slower but cheaperstorage (e.g. physical hard disks, tape drives, etc.).

Specific database applications or users also exhibit different workloadpatterns. Some make high-frequency access to data while others may makeless frequent data accesses. In another approach, workload managementtechniques perform a similar task to multi-temperature storage in thatthey provide a mechanism to give different workloads differential levelsof access to resources. For example, giving a high-priority workload alarger share of available CPU time than other workloads is similarqualitatively to giving a given subset of data residence on a fasterbacking storage device.

However, the above approaches require user intervention to determinewhich data or workloads should get proportionally more access to thefast and expensive resources, and which should be relegated more to theslower and cheaper resources.

SUMMARY

According to one embodiment of the present invention, a method forintermittent sampling of storage access frequency, implemented by acomputing processor, determines a duration of a collection window. Aduration of an observation window within the collection window is alsodetermined. A position of the observation window within the collectionwindow is randomly selected. Frequencies of accesses of one or morestorage objects during the observation window are observed.

In one aspect of the present invention, when a new access of a givenstorage object occurs, a delta time for the given storage object iscalculated as a current time of the new access minus a timestamp of amost recent observed prior access of the given storage object.

In one aspect of the present invention, the delta time of two sequentialaccesses of the given storage object in two different observationwindows is calculated as if the two different observation windows areimmediately adjacent to each other.

System and computer program products corresponding to theabove-summarized methods are also described and claimed herein.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a system according to embodiments ofthe present invention.

FIG. 2 illustrates a method for defining database objects for storage ina storage hierarchy according to embodiments of the present invention.

FIG. 3 illustrates a method for intermittent sampling of storage accessfrequency according to embodiments of the present invention.

FIGS. 4A and 4B illustrate examples of delta time calculation accordingto embodiments of the present invention.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java® (Java, and all Java-based trademarks and logos aretrademarks of Oracle Corporation in the United States, other countries,or both), Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified local function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

FIG. 1 illustrates an embodiment of a system according to embodiments ofthe present invention. The computer system 100 is operationally coupledto a processor or processing units 106, a memory 101, and a bus 109 thatcouples various system components, including the memory 101 to theprocessor 106. The bus 109 represents one or more of any of severaltypes of bus structure, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. The memory 101 mayinclude computer readable media in the form of volatile memory, such asrandom access memory (RAM) 102 or cache memory 103, or non-volatilestorage media 104. The memory 101 may include at least one programproduct having a set of at least one program code module 105 that areconfigured to carry out the functions of embodiment of the presentinvention when executed by the processor 106. The computer system 100may also communicate with one or more external devices 111, such as adisplay 110, via I/O interfaces 107. The computer system 100 maycommunicate with one or more database management systems (DBMS) 112 vianetwork adapter 108.

FIG. 2 illustrates a method for defining database objects for storage ina storage hierarchy according to embodiments of the present invention.The database objects may include tables, indexes, etc. of a database.The method observes the frequencies of accesses of a plurality ofdatabase objects over a predetermined time period (201). The methodcomputes the mean and standard deviation for the plurality of databaseobjects based on the observed frequencies of accesses of the databaseobjects (202). For a given database object, the method determines thez-score based on a comparison of the given object's frequency of accessto the mean and standard deviation computed for the plurality ofdatabase objects (203). In this embodiment, the z-score indicates thedifference, in units of the standard deviation, between the frequency ofaccess of the given database object and the mean and standard deviationfor all database objects being observed. The method then determines thelevel in the storage hierarchy corresponding to the z-score of the givendatabase object (204). The database objects may then be stored or movedto a level of the storage hierarchy corresponding to their z-scores. Inthis embodiment, when a new database object access is detected, themethod calculates the delta time (ΔT) as the current time minus thetimestamp of the most recent access of the database object. The ΔT isthen used to update cumulative statistics for the database object, whichincludes: a count of observations for the database object; the sum ofΔT's (ΣΔT) for the database object; and the sum of the squares of ΔT's(Σ(ΔT)²) for the database object. These three cumulative statisticsenable the computing of the mean and the standard deviation for thedatabase object, without having to save the timestamp of every access.

In one embodiment, the activities of the database are monitoredcontinuously. In some situations, this continuous monitoring negativelyimpacts performance and temporary storage for saving capturedinformation. To address the impact on performance and temporary storage,another embodiment according to the present invention monitors theactivities of the database intermittently. However, with intermittentmonitoring, missing periodic activity entirely may be possible due toharmonic aliasing. For example, if monitoring is enabled for the firstminute of every hour, and a job was scheduled to run for ten minutesevery four hours on the half hour, the monitor would never capture theinformation about accesses made by this job. In this embodiment, themethod avoids missing accesses due to harmonic aliasing by randomizingthe distribution of an observation window within larger collectionwindows, as described further below.

FIG. 3 illustrates a method for intermittent sampling of storage accessfrequency according to embodiments of the present invention. First, themethod determines a duration or size of a collection window (301). Themethod also determines a duration or size of an observation windowwithin the collection window (302). A position of the observation windowwithin the collection window is randomly selected (303). The method thenobserves the frequencies of accesses of database objects during theobservation window (304). In this embodiment, a user is allowed tochoose a percentage of time to enable the observation of storageaccesses. For example, assume that the duration of the collection windowis 100 minutes, i.e., every 100 minutes, an existing collection windowends and a new collection window begins. Assume also that the userchooses to observe 1% of the time. The observation window is thusdetermined to be one minute long (1% of 100 minutes). The methodrandomly selects a position of the one minute observation window withinthe 100 minutes of the collection window, during which database eventsare observed. In the rest of the collection window, observation isdisabled. For any database object accesses that occur during this oneminute observation window, ΔT is calculated for the database object asthe current time minus the timestamp of the most recent access of thedatabase object within the observation window, as set forth above. Inthis manner, database object accesses by regularly scheduled jobs may beobserved at least once. When observed over an extended period of time,the observed frequency of accesses of the database object will closelyapproximate the true frequency as if each access is observed.

Typically, the first and last data points of each observation window arediscarded. However, when the time between accesses of a database objectis large, discarding the first and last data points would prevent ΔTfrom being calculated accurately. For example, when a first access of adatabase object is within one observation window, and the nextobservation is not until another observation window, ΔT cannot becomputed in the same manner as above. Further, when the frequency ofaccesses is low, some observation windows may not observe any accesses.One embodiment of the present invention addresses this scenario bysubtracting the amount of time observations are disabled between twoobservation windows containing two sequential access observations forthe database object. For example, FIG. 4A illustrates two observationwindows W1 and W2. Observations are disabled for time period D1 betweenW1 and W2. Assume in this example that a first observation of an accessto a database object occurs at time T1 in observation window W1. Assumealso that the next observed access to the database object occurs at timeT2 in observation window W2, with no observations in-between. In thisscenario, ΔT is calculated as equal to T2−T1−(width of D1). Time periodD1, in which observation is disabled, is removed from the calculation ofΔT, and ΔT is calculated as if the two observation windows, W1 and W2,are immediately adjacent to each other.

For another example, FIG. 4B illustrates three observation windowsW1-W3. Observations are disabled for time periods D1 and D2. Assume inthis example that a first observation of an access to a database objectoccurs at time T1 in observation window W1. Assume also that the nextobserved access to the database object occurs at time T2 in observationwindow W3. No accesses to the database object are observed inobservation window W2. In this scenario, ΔT is calculated as equal toT2−T1−(width of D1)−(width of D2). The time periods, D1 and D2, in whichobservation is disabled, are removed from the calculation of ΔT. Notethat the width of the observation window W2 is not subtracted from ΔT,even though there were no observed accesses, because W2 represents timeduring which observation was enabled and thus a lack of accesses for theduration of W2 represents information about the true frequency ofaccesses to the database object. In this manner, for low frequencyaccesses, ΔT can still be calculated.

In some database systems, an event monitor is used to monitor theactivities of the database and used to observe accesses to databaseobjects. The event monitor may be implemented with two switches, onestate switch that enables and disables the monitor itself, and anotherswitch in the form of a collect clause for a database workload. Forperformance reasons, activities would not collect monitoring informationunless the collect clause is effective on the database workload at thestart of the workload, and this information would not go to the eventmonitor unless the event monitor itself is enabled at the end of theworkload. However, when both the state and the collect clause aretoggled to indicate the beginning and the end of an observation window,long-running activities that start before the beginning of theobservation window and/or end after the end of the observation windowmay never be observed. To address this scenario, one of the switches maybe maintained in the “on” or enabled position at all times. In thismanner, whether a given access will be observed depends only on a singlecheck at a single point in time for that access. If that single checkfalls within the observation window, the access will be observed.

Although embodiments of the present invention are described above in thecontext of database objects, any storage object may be intermittentlysampled in the above manner without departing from the spirit and scopeof the present invention. Furthermore, the intermittency may be temporaland/or spatial in nature. An example of spatial intermittency iscalculating the spatial frequency of occurrence of an item of interestwhen observations are made for several non-adjacent segments of spaceinstead of time as in the prior examples. In such a case the axeslabeled “Time” in FIG. 4A and FIG. 4B can be relabeled “Distance” and inall other respects the same calculations can be applied as in theoriginal examples.

The descriptions of the various embodiments of the present invention hasbeen presented for purposes of illustration, but are not intended to beexhaustive or limited to the embodiments disclosed. Many modificationsand variations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1-7. (canceled)
 8. A computer program product for intermittent samplingof storage access frequency, the computer program product comprising: acomputer readable storage medium having computer readable program codeembodied therewith, the program code executable by a processor to:determine a duration of a collection window; determine a duration of anobservation window within the collection window; randomly select aposition of the observation window within the collection window; andobserve frequencies of accesses of one or more storage objects duringthe observation window.
 9. The computer program product of claim 8,wherein the program code executable by the processor to observe thefrequencies of the accesses of the one or more storage objects isfurther executable by the processor to: determine that a new access of agiven storage object occurs; and in response to determining that the newaccess of the given storage object occurs, calculate a delta time forthe given storage object as a time of the new access minus a timestampof a most recent observed prior access of the given storage object. 10.The computer program product of claim 9, wherein the program codeexecutable by the processor to calculate the delta time as the time ofthe new access minus the timestamp of the most recent observed prioraccess of the given storage object is further executable by theprocessor to: calculate the delta time of two sequential accesses of thegiven storage object in two different observation windows as if the twodifferent observation windows are immediately adjacent to each other.11. The computer program product of claim 10, wherein the program codeexecutable by the processor to observe the frequencies of the accessesof the one or more storage objects is further executable by theprocessor to: observe the frequencies of the accesses of the one or morestorage objects in a first observation window (W1) and a secondobservation window (W2), wherein observation is disabled during a timeperiod (D1) between W1 and W2; observe a first access of the givenstorage object at a first time (T1) in W1 and a second access of thegiven storage object at a second time (T2) in W2, wherein no access ofthe given storage object is observed between T1 and T2; and calculatethe delta time as equal to T2−T1−(width of D1).
 12. The computer programproduct of claim 10, wherein the program code executable by theprocessor to observe the frequencies of the accesses of the one or morestorage objects is further executable by the processor to: observe thefrequencies of the accesses of the one or more storage objects in afirst observation window (W1), a second observation window (W2), and athird observation window (W3), wherein observation is disabled during atime period (D1) between W1 and W2 and during a time period (D2) betweenW2 and W3; observe a first access of the given storage object at a firsttime (T1) in W1 and a second access of the given storage object at asecond time (T2) in W3, wherein observation is disabled during D1 andD2, wherein no access of the given storage object is observed between T1and T2; and calculate the delta time as equal to T2−T1−(width ofD1)−(width of D2).
 13. The computer program product of claim 8, whereinthe program code executable by the processor implements an event monitorto observe the frequencies of the accesses of the one or more storageobjects, wherein a monitoring by the event monitor is enabled anddisabled using a state switch for the event monitor and a collect clausefor a database workload, wherein either the state switch or the collectclause are maintained in an enabled position.
 14. The computer programproduct of claim 9, wherein the program code is further executable bythe processor to: compute a mean and standard deviation for a pluralityof storage objects based on observed frequencies of accesses of theplurality of storage objects; determine a z-score for the given storageobject based on a mean computed over the delta times for the givenstorage object and the mean and standard deviation of the plurality ofstorage objects; and determine a level in the storage hierarchycorresponding to the z-score of the given storage object.
 15. A system,comprising: a processor; and a computer readable storage medium havingcomputer readable program code embodied therewith, the program codeexecutable by a processor to: determine a duration of a collectionwindow; determine a duration of an observation window within thecollection window; randomly select a position of the observation windowwithin the collection window; and observe frequencies of accesses of oneor more storage objects during the observation window.
 16. The system ofclaim 15, wherein the program code executable by the processor toobserve the frequencies of the accesses of the one or more storageobjects is further executable by the processor to: determine that a newaccess of a given storage object occurs; and in response to determiningthat the new access of the given storage object occurs, calculate adelta time for the given storage object as a time of the new accessminus a timestamp of a most recent observed prior access of the givenstorage object.
 17. The system of claim 16, wherein the program codeexecutable by the processor to calculate the delta time as the time ofthe new access minus the timestamp of the most recent observed prioraccess of the given storage object is further executable by theprocessor to: calculate the delta time of two sequential accesses of thegiven storage object in two different observation windows as if the twodifferent observation windows are immediately adjacent to each other.18. The system of claim 17, wherein the program code executable by theprocessor to observe the frequencies of the accesses of the one or morestorage objects is further executable by the processor to: observe thefrequencies of the accesses of the one or more storage objects in afirst observation window (W1) and a second observation window (W2),wherein observation is disabled during a time period (D1) between W1 andW2; observe a first access of the given storage object at a first time(T1) in W1 and a second access of the given storage object at a secondtime (T2) in W2, wherein no access of the given storage object isobserved between T1 and T2; and calculate the delta time as equal toT2−T1−(width of D1).
 19. The system of claim 17, wherein the programcode executable by the processor to observe the frequencies of theaccesses of the one or more storage objects is further executable by theprocessor to: observe the frequencies of the accesses of the one or morestorage objects in a first observation window (W1), a second observationwindow (W2), and a third observation window (W3), wherein observation isdisabled during a time period (D1) between W1 and W2 and during a timeperiod (D2) between W2 and W3; observe a first access of the givenstorage object at a first time (T1) in W1 and a second access of thegiven storage object at a second time (T2) in W3, wherein observation isdisabled during D1 and D2, wherein no access of the given storage objectis observed between T1 and T2; and calculate the delta time as equal toT2−T1−(width of D1)−(width of D2).
 20. The system of claim 16, whereinthe program code is further executable by the processor to: compute amean and standard deviation for a plurality of storage objects based onobserved frequencies of accesses of the plurality of storage objects;determine a z-score for the given storage object based on a meancomputed over the delta times for the given storage object and the meanand standard deviation of the plurality of storage objects; anddetermine a level in the storage hierarchy corresponding to the z-scoreof the given storage object.